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Abstract 



This article reports the results of a meta-analysis of 20 research articles containing 89 effect sizes 
related to the use of digital tools and learning environments to enhance literacy acquisition. 
Results (weighted effect size of 0.489) demonstrate that technology can have a positive effect on 
reading comprehension, but little research has focused on the effect of technology on 
metacognitive, affective, and dispositional outcomes. We conclude that although there is reason 
to be optimistic about using technology in middle-school literacy programs, there is also reason 
to encourage the research community to redouble its emphasis on digital learning environments 
for students in this age range and to broaden the scope of the interventions and outcomes they 
study. 
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Background for the Meta- Analysis 

Literacy, and reading in particular, is just one of the many areas in which research has provided 
evidence of the potential impact of new technologies such as multimedia and hypermedia. 
Unfortunately, most of the studies in this research corpus have addressed literacy or reading 
acquisition in the early years of schooling. These technologies may be equally as important for 
older readers, particularly those who have not experienced great success in their school careers. 
To assess our collective and cumulative knowledge about the impact of digital tools on the 
reading performance of middle-school students, we conducted a meta-analysis. 

The primary purpose of this work was to determine whether digital technologies can affect the 
acquisition of advanced reading skills, such as comprehension, metacognition, strategy use, and 
motivation and engagement. Another purpose was to identify, or at least to point in the direction 
of, substantive (i.e., topics or skills are being taught), technical, and contextual factors associated 
with effective interventions. The ultimate outcomes of this second purpose, we hoped, would be 
a set of implications to guide policy makers in their quest to improve reading acquisition in these 
vexing middle- school years and a menu of promising pathways to guide future research. 

The Evolving Relationship Between Literacy and Technology 

Literacy and technology are two words that seem to be increasingly paired in today’s worlds of 
research, practice, and policy. People often describe the need to become computer literate; 
authors write about digital literacy (and related terms such as visual literacy and media literacy) 
as one of the important new discourses in our schools; and research has investigated the role of 
technology in improving literacy acquisition and instruction. 

The first of these issues, the need to become computer literate, is very real in the policy and 
practice of today’s schools. The National Educational Technology Standards (NETS), for 
instance, have been developed to ensure that children are learning with technology and using 
digital tools to acquire knowledge in content areas (ISTE, 2005). The International Reading 
Association suggested the following rights in a 2001 position statement on literacy and 
technology: 

• Teachers who are skilled in the effective use of Information Communications 
Technology (ICT) for teaching and learning 

• A literacy curriculum that integrates the new literacies of ICT into instructional programs 

• Instruction that develops the critical literacies essential to effective information use 

• Assessment practices in literacy that include reading on the Internet and writing using 
word-processing software 

• Opportunities to learn safe and responsible use of information and communication 
technologies 

• Equal access to ICT 
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Such goals and standards include not just attaining comfort with and knowledge of the machine, 
but also related literacies including information literacy, visual literacy, digital literacy, new 
literacy, critical literacy and media literacy (Holum & Gahala, 2001). 

As one looks broadly at the interface of technology and literacy, perhaps most potentially 
rewarding for literacy educators is the role of technology in literacy acquisition and instruction, 
especially for primary grade populations. We know, for example, that electronic storybooks help 
improve student comprehension and motivation (Matthew, 1997; Doty, Popplewell, & Byers, 
2001) and that they also provide immediate decoding feedback to students (Labbo & Kuhn, 

1998; deJong & Bus, 2002; Cazet, 1998; Doty, Popplewell, & Byers, 2001). 

In addition to electronic storybooks, teachers use software such as KidPix (Labbo, Eakle, & 
Montero, 2002), HyperStudio, and Microsoft PowerPoint to help students leam to decode. 
Websites such as Hot Potatoes (http://web.uvic.ca/hrd/halfbaked/) and Enchanted Learning 
(http://www.enchantedleaming.com/Home.html) provide cloze exercises and paragraph, 
sentence, and letter scramblers. PBS Kids & Sesame Street’s Letter of the Day 
(http://pbskids.org/sesame/letter/), Scholastic’s Letter Match (http://teacher.scholastic.com/ 
Clifford 1/flash/confusable/). Even Merriam-Webster’s allegedly lexicographically oriented 
website (http://www.m-w.com/) provides support for phonemic awareness and instruction. 
Finally, Leu & Kinzer (1999) have argued that (1) Internet activities, (2) Internet projects, 

(3) Internet inquiries, and (4) Internet workshops can lead to effective literacy instruction and 
reading comprehension. 

Technology is also used for writing instruction; indeed, the interface of technology and writing is 
sufficiently sophisticated to have attracted both “best practice” syntheses as well as meta- 
analysis (Goldberg, Russell & Cook, 2003). The Venn Diagram website 
(http://www.venndiagram.com/), software tools such as Inspiration and Microsoft PowerPoint, 
and hardware like Smartboards & Interactive Whiteboards provide students with opportunities to 
create concept maps and Venn diagrams to organize their writing. E-zines, or electronic 
magazines, not only provide current and authentic reading material for students, they also 
publish student work and thus act as an authentic audience for student writing. Electronic 
portfolios are providing ways for students to showcase their writing to teachers, other students, 
and parents. 

Even simple word processors have ‘tracking changes’ features where students can collaborate in 
their writing and thus receive scaffolding in their development. Blogs can provide online 
journaling space for students to write about their growing expertise or their daily observations 
(Ferdig & Trammell, 2004) and word searches, word games, and online dictionaries and 
thesauruses build students vocabulary and confidence in language use. Students and teachers also 
find great writing practice using webquests and inquiry pages. Finally, students get writing 
practice through authentic projects such as Keypals, where they write with classrooms in 
different states or countries, and the Internet Project Registry, where classes can register their 
projects and collaborate with students from around the world. 

Beyond reading and writing, technology has been used to increase access to images of and 
information about diversity in classrooms, both at the student level with projects like I Love 
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Languages (http://www.ilovelanguages.com/) and Say Hello to the World 
(http://www.ipl.org/div/kidspace/hello/) and also at the instructor and preservice level with 
projects like CTELL (Teale, Leu, Labbo, & Kinzer, 2002) and The Reading Classroom Explorer 
(Ferdig, Roehler, & Pearson, in press). Technology has been used to give struggling readers 
access to scaffolding and individualized instruction through projects like Technology-Enhanced 
Literacy Environment-Web (TELE-Web; Zhao, Englert, Jones, Chen, & Ferdig, 2000). 
Computers and even older media such as audio and video recorders give students practice with 
spoken language. Free online archives provide reading material for both storytelling and 
literature classes. Finally, online journals, listservs, discussion forums, and associations provide 
continued professional development for the literacy instructor. 

In short, we have witnessed a proliferation of applications of various sorts of technology for 
various populations of users from preschoolers to teachers. But have we conducted enough 
careful research in technology education field to have reached a point of maturity sufficient to 
merit extensive reviews, such as best-evidence syntheses and meta-analyses of various aspects of 
technology tools? 

We have certainly made those attempts in recent years, but with varying degrees of success. In 
recent years, Cavanaugh et al. (2004) provided evidence in their meta-analysis that distance 
education is as effective as face-to-face classroom instruction. Shachar & Neumann (2003) 
analyzed studies on distance learning and found that distance learners outperformed counterpart 
students in face-to-face classrooms in two thirds of the studies. Waxman et al. (2003) 
synthesized the literature on teaching and learning with technology and found it had a positive 
significant effect on student outcomes when compared to instruction without technology, a 
finding supported by others in this field (Kulik & Kulik, 1991; Kulik & Kulik, 1986). 

Turning to the purview of the present study, there have been a few recent meta-analyses related 
to literacy and technology. Goldberg, Russell, & Cook (2003) synthesized 26 studies from 1992- 
2002 and found that computers improved the quality and quantity of writing compared to 
classrooms without technology. They did find mixed results, however, for revision behaviors. 
Torgerson, Porthouse, & Brooks (2003) found a modest benefit for computer-assisted instruction 
for literacy acquisition of imprisoned adults, but it was not statistically significant. Finally, 
Torgerson & Elboume (2002) completed a meta-analysis on the effects of Information 
Communications Technology on spelling, finding what they characterize as a modest but not 
statistically significant effect favoring technology in the teaching of spelling. 

On the specific question of the empirically established relationships between literacy and 
technology, Leu (in press) has suggested that our scholarship to date warrants at least three 
distinct conclusions: (a) Technology is transformative, changing the nature of literacy (see also 
Reinking, 1998); (b) the relation between literacy and technology is transactional (see Bruce, 
1997); and (c) technology is deictic, which means that it will change rapidly in response to 
environmental forces. 

Unfortunately, what is less clear is what the research can definitively suggest about the relation 
between technology and literacy. One problem is the alarmingly low number of published 
research studies investigating technology and literacy (Leu, in press; Kamil & Lane, 1998). 
Clearly either more research has to be done or we need a better approach to identifying and 
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analyzing relevant existing research. The current endeavor is predicated on the assumption that 
although we may well need more and better research, it is time to take stock of what we do 
know, if for no other reason than to highlight gaps to guide the field in future scholarly efforts. 
And, if the effects we do find are truly powerful, even though limited in scope, we should 
publicly acknowledge and use what we do know and can recommend to policymakers with 
confidence. 

Concerns About Literacy, Technology, and Adolescents 

No Child Left Behind (NCLB) funds reading programs (Reading First) that focus primarily on 
Prekindergarten through Grade 3; however, the NCLB Act (2002) also requires that students in 
Grades 4 through 12 make adequate yearly progress toward meeting state reading standards. 
Additionally, the Reading First provision of NCLB dictates that students who are not making 
adequate progress in the middle-school years be offered research-based interventions to 
accelerate their learning. Finally, even though the lion’s share of the resources for improving 
reading in the context of current policy go to the primary grades, the rhetoric about the need for 
focusing greater attention and resources on adolescent literacy has been steadily mounting for the 
past few years. 

Several professional organizations, in fact, have championed this shift in attention. For example, 
the National Reading Conference (NRC) commissioned a white paper on Effective Literacy 
Instruction for Adolescents (Alvermann, 2001) that explicitly acknowledges the complexities of 
reading in relation to writing and oral language in an array of 21st-century media environments, 
including, of course, print. The International Reading Association, in its Position Statement on 
Adolescent Literacy (2002), echoed this perspective by emphasizing the importance of (a) access 
to a wide variety of reading materials, (b) building skills and desire to read complex materials, 

(c) modeling and giving explicit instruction, and (d) developing an understanding of the 
complexities of individual adolescent readers. 

While our empirical knowledge may be weak, individuals have used theoretically based 
arguments, grounded in best practice and compelling cases, to draw conclusions about the degree 
to which technology tools can and do support literacy teaching and learning for adolescents. 
Although Alvermann (2001) cites little empirical research on the topic generally, and even less 
that applies specifically to instruction at the middle and high school levels, she, along with 
others, provides relevant examples to illustrate how adolescents are making valuable reading- 
writing connections in their bid to communicate in a computer-mediated world (e.g., Beach & 
Bruce, 2002; Beach & Lundell, 1998; Homey & Anderson-Inman, 1994). 

Other work suggests that American youth are turning more and more toward the Internet as their 
primary textbook and spend more time with media than any other single activity (Gee, 2002; 
Lenhart, Simon, & Graziano, 2001; Levin & Arafeh, 2002). Levin and Arafeh (2002) found, for 
example, that 71 percent of students pointed to the Internet as their primary resource for 
completing homework assignments. These same students actually regarded the Internet as more 
relevant to their daily lives than other forms of information, a finding suggesting that schools are 
woefully slow on the Internet uptake. We agree with O’Brien (in press) that the widespread use 
of the Internet and other digital tools among youth requires educators to facilitate students’ 
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experiences with digital literacy tools in school. What we are less certain about, and certainly 
less knowledgeable about, is the particular focus that facilitative support should take. Indeed, the 
fact that so many scholars of adolescent literacy resort to compelling cases to support their policy 
and practice recommendations about literacy underscores the need for precisely the sort of 
synthesis we have undertaken. 

The Context for the Synthesis 

Commissioned by the North Central Regional Educational Laboratory (NCREL) Center for 
Technology, in collaboration with the NCREL Center for Literacy, we conducted a research 
synthesis of experimental and quasi-experimental studies, conducted within the last decade and a 
half of the start of the project (1988 was set as the cut off), that focus on interventions using 
digital literacy tools to improve the reading performance of middle school students. To narrow 
the research synthesis topic from the original and broad topic of “the effectiveness of technology 
on student achievement in literacy” found in NCREL’ s 2004 Updated Annual Plan, we looked at 
the What Works Clearinghouse (WWC) topic areas to ensure that we were not duplicating efforts 
in place by WWC developers. We considered topics that could contribute to the WWC as well as 
continue to inform the work of NCREL’ s Center for Technology and Center for Literacy for the 
foreseeable future; hence the current emphasis on the impact of digital tools on adolescent 
reading. 

Specifically, our research synthesis set out to use the tools of meta-analysis to answer questions 
about five areas designated in the request for proposals from the funding agency on the grounds 
that their answers would provide information essential to improved reading performance for 
adolescents: the impact of digital literacy tools on middle-school students in the following areas: 

• Strategy use 

• Metacognition 

• Reading motivation 

• Reading engagement 

• Reading comprehension 

We sought studies that attempted to both improve and measure progress in one or more of these 
areas. We defined digital literacy tools broadly to include a wide range of the use of media 
forms — images, video and audio clips, hypertext, hypermedia, Web pages, learning 
environments, and particular formats of presenting information for student learning. Of particular 
interest were the media forms of hypertext, hypermedia, and Web pages; we hoped that we 
would find a substantial body of experimental and quasi-experimental work examining these 
particular forms. This focus was strategic and intentional. We knew that the concepts of 
hypertext and hypermedia are considered crucial to understanding the interactions between 
reader and text in a multimedia environment. Additionally, the conventional wisdom about the 
effect of hypertext and other media on reading performance, especially in content area reading, is 
optimistic and enthusiastic (Vacca & Vacca, in press). We wanted to know whether such a high 
level of enthusiasm is supported by the available evidence. 
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Method 



Inclusion Criteria 

A study was included in this meta-analysis if it met the following criteria: 

• Was subjected to a peer review process. This excluded studies such as doctoral 
dissertations, conference presentations, and unpublished reports, but it did include 
prepublication project reports that were peer-reviewed. 

• Included students in the middle grade school levels (6th, 7th, and 8th grade). Those 
studies that only reported results on these levels were labeled “right on target.” There 
were studies that included earlier or later grades along with the middle level grades. 
Where possible we only used the effect size data from the target grade levels. 
Occasionally when data could not be disaggregated (e.g., Grades 5-7 were lumped 
together), we spilled over into adjacent grade levels. 

• Used technology as the independent or moderating variable in the examination of reading 
skills. 

• Reported outcomes assessing the impact of a treatment on reading comprehension, 
metacognition, strategy use, and/or motivation. 

• Used an experimental or quasi-experimental design, including pretest-posttest designs. 

• Reported sufficient statistics to permit the calculation of an effect size. 

• Was published between 1988-2005. The time period was decided upon to address articles 
that had not been reviewed in previous and broader meta-analyses on the relationship 
between technology and reading processes. 

Location and Selection of Publications 

In an effort to be inclusive (and to take advantage of work conducted around the world) the 
search process was purposefully broadened to include studies from as many countries, languages, 
and cultural ranges as possible. We searched and included studies from many geographical areas 
as well as studies written in Spanish (one of the authors is a native Spanish speaker). It should be 
noted that most of the international journals consulted publish in English. We found a few 
candidate studies in Spanish, one of which survived into the final pool; many candidates and 
several finalists came from research conducted outside North America. 

An exhaustive search of databases, journals, websites, and bibliographic resources was carried 
out for studies that could even plausibly meet the established inclusion criteria. Five main 
searches were completed. First, drawing on various combinations of keywords (Appendix A), 
web searches were performed using such search engines as Google, Google Scholar, Yahoo, 
Metacrawler, Search.com, AskJeeves, AltaVista and Fycos. Second, similar keywords were used 
to systematically search academic and educational databases (Appendix B). The third search 
method was to examine abstracts in 79 educational technology, special education, psychology, 
literacy, and reading journals both in print and electronic modes, as not all journals or issues are 
available electronically (Appendix C). Fourth, in an effort to cover other cultural and linguistic 



Learning Point Associates 



The Effects of Technology on Reading Performance in the Middle-School 
Grades: A Meta- Analysis With Recommendations for Policy — 7 




ranges, abstracts in 34 relevant international journals were searched (Appendix D). Finally, Web 
sites of several reading and education professional organizations and research institutes were 
browsed for studies. Examples of such sites are the various regional educational laboratories, the 
Center for Research on Evaluation, Standards, and Student Testing, the research centers for 
various states departments of education, the Rand Corporation, and federal institutes such as the 
National Institute of Child Health and Human Development. 

The Filtering Process for the Selection of the Target Articles 

The initial strategy for this search process was extensive rather than intensive. The goal was to 
identify the maximum number of studies and articles that met or even came close to meeting the 
inclusion criteria. “Backward mapping” was also used; we consulted the references at the end of 
target articles for potential other studies. Finally, literacy and technology experts (operationally 
defined as individuals whose works we encountered searching the literature, both in the U.S. and 
abroad, were contacted to solicit advice and information on studies not found in the searches or 
in the journals examined. 1 

From this initial filtering stage, 204 full-text candidate articles or reports were obtained and 
evaluated for inclusion in the analysis. These 204 articles were subjected to a second screening 
that included four key criteria. First, the type of study was examined to determine if the study 
was experimental, quasi-experimental, a natural experiment, a literature review, a correlation 
study, or a qualitative study. We were interested only in experimental or quasi-experimental 
studies as is appropriate and necessary for conducting a meta-analysis. Second, the grade level of 
the subjects were coded; only articles that had at least some subjects in Grades 6-8 were 
included. Next, articles were included if the content of the study was at least partially related to 
reading (rather than writing, language arts, or some other content interest), in terms of the 
intervention and the outcome. Finally, articles were coded for the use of the technology in the 
study. Articles were not included in the meta-analysis if technology was not used or if the use of 
technology was incidental. Each article was read by at least two of the four authors of the report. 
When we applied these four criteria, our sample was reduced from 204 to 38. 

The third filtering stage occurred when we used the criteria from the coding manual developed 
by Waxman and his colleagues (2003) for NCREF. In the process of applying these criteria, 
which included the computation of effect sizes for each dependent measure, the set of articles 
was trimmed to the 20 that eventually were used in the meta-analysis. Several studies did not 
have sufficient data to compute the effect size or had reported results without control or 
treatment statistics. The complete Waxman-derived codebook is located in Appendix E. For both 
the second and third filters, we consulted the criteria included in the “Study Review Standards” 
from the What Works Clearinghouse (What Works Clearinghouse, n.d.); in fact, many of the 
standards in the third (i.e., Waxman, 2003) filter are identical to those proposed by 



1 See Appendix G for a full bibliography of studies reviewed. 
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Clearinghouse. Table 1 provides a summary of the major categories and the variables that were 
examined in depth in this meta-analysis. 



Table 1. Description of the Coding Characteristics 
With a Brief Description of Levels and Findings 



Major 

Category 


Brief Description of the 
Major Category 


No. of 
Variables 


Variables Examined In-depth 


Study 

Characteristics 


This category contained descriptive 
information about the study. It 
included the name, year, and author(s) 
of the article. It also included 
variables like gender, country, region, 
ethnicity, and target audience. 


17 


Author 

Year 

# of comparisons 
Student sample size 
Journal of Publication 
Target Population 


Study of 

Quality 

Indicators 


Variables within this category related 
to the factors helping determine the 
quality of the study. These variables 
included the name of the measure and 
its reliability, the pretest equivalency, 
and various outcomes 


12 


Duration of study 
Cognitive outcomes 
Affective outcomes 
Behavioral outcomes 
Effect size coefficient 
Weight 


Sources of 
Invalidity 


History, maturation, selection bias, 
type of design, and selection- 
maturation interaction are all 
examples of sources of invalidity that 
were coded in this category. 


14 


The sources of invalidity in the 
codebook provided a way to 
examine whether the 
methodologies provided in the 
studies were rigorous enough to 
include the results in the rneta- 
analysis. As such, all 14 variables 
were examined to help filter the 
selected corpus of articles. 


Reading 

Characteristics 


The reading characteristics category 
included variables to describe both 
the focus of the intervention (what 
they did) and the outcome of the 
intervention (what they observed). 


2 


Examples of potential codes for 
the two variables included: 

Phonics 

Phonemic awareness 
Vocabulary 

Reading comprehension 
Reading Volume 
Reader response 
Fluency 

Independent reading 
Meta-cognition 
Content Learning 
Spelling 

Word Recognition 
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Major 

Category 


Brief Description of the Major 
Category 


No. of 
Variables 


Variables Examined In-depth 


Technology 

Characteristics 


The technology characteristics 
category examined the technology 
features of the study. Variables 
included the type of technology used, 
the role or focus of the technology, 
and the teacher and students’ prior 
experience with technology. 


19 


Aegis of technology 


Instructional / 

Teaching 

Characteristics 


Instructional and teaching 
characteristics were examined in this 
category. Examples include the 
setting and mode of instruction, 
collaboration, and what types of 
conversations where encouraged in 
the pedagogy. 


7 


Unfortunately, in many cases, this 
information was not clearly 
delineated in the research article. 
Therefore, no information was 
gathered from the 20 articles to 
run meaningful analyses for this 
category. 


Policy 


The final category related to the 
policy focus of the study. This 
category contained two variables: the 
level of policy (i.e. state or national) 
and the policy focus. 


2 


Not enough information was 
included in most articles to 
analyze the level of policy. The 
policy focus was examined and 
included the following 
possibilities: 

Unspecified 

Reducing achievement gaps 
Increased use of technology 
Increased specific type of use 
Improve Specific Educational 
Outcomes 



Statistical Treatment 

To obtain effect sizes, the quantitative results from individual studies were transformed into a 
standardized difference between the treatment and the control group on a given measure. We 
calculated the effect sizes by taking the mean performance difference between the group that 
received technology experimental treatment and the control group and dividing it by a pooled 
standard deviation. Because it has been documented that this effect-size index tends to be 
upwardly biased when based on small sample sizes. Hedges’ (1981) correction was applied. 

The Hedges correction uses the inverse variance weight to give more weight to studies with 
larger sample sizes. The Hedges “g” statistic, a weighted effect-size estimate, was used in all 
subsequent analyses. Two different effect-size calculation methods were utilized depending on 
the summary statistics reported within the individual research studies: posttest means and 
standard deviations (n= 16) and between-groups independent 1 test (n= 4). Effect sizes were 
computed using formulas provided by Lipsey & Wilson (2001); in a few instances, we used t or 
Ftest statistics to infer appropriate values. 
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Selecting a Model 

The statistical models for meta-analysis are broadly classified into two types: fixed effects and 
random effects. Fixed-effects models generalize to a hypothetical population of studies, from 
which one assumes to have drawn a random sample. Random-effects models generalize to a 
population of subjects. The models differ in the way they treat the variability of the results 
between the studies. 

The fixed-effects model treats variability as exclusively due to random variation; thus if all the 
studies were infinitely large they would give identical results. The random effects model assumes 
a different underlying effect for each study and takes this into consideration as an additional 
source of variation. In general, random effects models are more conservative because they result 
in wider confidence intervals than the fixed-effects model. In all of our analyses we used the 
random-effects model particularly due to the small number of studies and the related issues of 
homogeneity. 

Three types of data analyses were performed: 

1. For each study, an independent set of effect sizes were extracted, weighted, and then 
aggregated. Using the combined effect size extracted from each study, an overall effect 
size across studies was calculated and tested for statistical significance. 

2. Analyses were performed to investigate heterogeneity and publication bias of the effect 
sizes. We utilized homogeneity testing and forest plot depiction. 

3. Based on our substantive interests in this area of research, we conducted several 
comparisons of the extent to which study features (e.g., population served or instructional 
focus) moderated the effect on outcome measures. For these comparisons we used the 
total of 89 effect sizes. In doing each of these specific comparisons, we computed a Q 
statistic to test the difference between effect sizes aggregated for the levels of a given 
variable (after Lipsey & Wilson, 2001). 

We used the weighted average, as recommended in the statistical literature to give more weight 
to larger studies with less random variation than to smaller studies. The method we used was the 
inverse variance method where the weights are equal to the inverse of each study’s estimated 
effect size. 

Computing Effect Sizes from Correlated Designs 

A consistently vexing question for those who undertake meta-analyses is how to compute effect 
sizes when there are correlated designs such as matched groups, repeated measures, within- 
subjects factorial design, and single subject among others. In these designs there are two 
possibilities to compute the effect size (ES) for a study. One possibility is to use the original 
standard deviations for the means of two groups (treatment and control). Another possibility is to 
take into account the correlation between two scores. If we follow the second possibility the 
calculated effect will be larger than the first possibility (at least when the correlation exceeds 
0.5). In our meta-analysis we did explore the calculations for both possibilities. However, the 
work done by Dunlop, Cortina, Vaslow, and Burke (1996) and Morris and DeShon (2002) 
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convincingly argues that original standard deviations should be used to compute ES for 
correlated designs. These authors argued and showed that if the pooled standard deviation is 
corrected for the amount of correlation between the measures, then the ES estimate will be an 
overestimate of the actual ES. (For those who are skeptical about this approach, it will be useful 
to note that we did actually use both approaches and found that when the effect sizes are 
calculated by taking the correlations into account, none of the major findings and conclusions are 
altered.) 
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Results and Discussion 



Descriptive Results 

The most striking result of the analysis is that we were able to locate data allowing us to address 
adequately only one of the five areas of reading about which we sought empirical evidence: 
comprehension. We found only two studies that provided outcome measures for strategy use 
(Solomon et al., 1999; Reinking, 1988). Interestingly in the Solomon study, the effect sizes were 
“off the charts” in favor of the technological training over the control group (which received no 
metacognitive emphasis); by contrast, the effect sizes in the Reinking study were 
inconsequentially in favor of the control group on strategy use (in the -0.05 to -0.10 range). We 
could not disentangle strategy use from metacognition (since strategy use is inherently 
metacognitive) either as an outcome or as an intervention focus, so in the final analysis we 
grouped them together. Four studies “mentioned” motivation but only two (Kramarski, 2000; 
Reinking, 1988) included measures of it, and engagement was only reported as a qualitative 
outcome by a few authors, most often to describe the delight teachers or students took in using 
the technology. 

The overwhelming emphasis was on reading outcomes, with comprehension as the most 
common of all outcome measures (65 percent); vocabulary, which we viewed as a member of the 
comprehension family, was a distant second, accounting for 10 percent of the outcomes. In terms 
of the emphasis of the interventions, the distribution was much more even than for outcomes. 
Most interventions attended to more than one aspect of reading; hence the highest incidence was 
for “mixed” emphases at 30 percent of the cases; for example, an environment would offer a 
hypertext learning environment with access to word pronunciation, word meaning, contextual 
information, and comprehension scaffolds to guide an individual’s reading. 

One might speculate that those who work in this medium are attempting to take full advantage of 
its capabilities. Among the single emphasis programs, the focus was fairly evenly distributed 
among vocabulary (17 percent), word recognition (15 percent), independent reading (12 percent), 
and comprehension instruction (12 percent). The intervention codings were aggregated to create 
two categories: a meaning emphasis (mixed, comprehension, vocabulary, metacognition, and 
independent reading) and code emphasis (word recognition, phonemic awareness, and fluency). 
Of the 20 studies, 15 were categorized as meaning emphasis, with only 1 clearly as code 
emphasis, and 3 categorized as other. 

Analysis of Effect Sizes 

The effect sizes (using the Hedges g correction for sample size) for all 89 outcomes are reported 
in Appendix F and summarized (as averages weighted for the number of effect sizes in each 
study) in Table 2 for the 20 studies that survived all three screens. 
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Table 2. Data on the Mean Weighted Effect Sizes for Each Study 





95 Percent 
Confidence 
Interval 






Number 
of Effects 


Hedges' 

g 


Standard 

error 


Variance 


Lower 

limit 


Upper 

limit 


Z- Value 


Alfassi 


3 


0.815 


0.352 


0.124 


0.125 


1.506 


2.314* 


Dalton 


1 


0.424 


0.204 


0.042 


0.023 


0.825 


2.075* 


Gentry 


3 


0.135 


0.279 


0.078 


-0.412 


0.682 


0.483 


Fasting 


6 


0.584 


0.284 


0.080 


0.028 


1.140 


2.059* 


Hasselbring 


4 


0.521 


0.181 


0.033 


0.166 


0.875 


2.876** 


Henao 


4 


0.668 


0.451 


0.203 


-0.215 


1.552 


1.483 


Higgins 


1 


0.600 


0.261 


0.068 


0.089 


1.111 


2.301* 


Jones 


10 


0.334 


0.193 


0.037 


-0.044 


0.712 


1.731 


Kramarski 


3 


-0.204 


0.283 


0.080 


-0.758 


0.350 


-0.721 


Leu 


6 


0.503 


0.303 


0.092 


-0.090 


1.097 


1.662 


Ligas 


8 


0.029 


0.093 


0.009 


-0.153 


0.210 


0.312 


Liu 


3 


2.679 


0.361 


0.130 


1.971 


3.387 


7.420** 


Reinking 8 8 


12 


0.214 


0.251 


0.063 


-0.278 


0.706 


0.852 


Reinking90 


7 


0.691 


0.371 


0.138 


-0.036 


1.419 


1.863 


Rouse 


4 


0.060 


0.136 


0.018 


-0.206 


0.326 


0.442 


Solomon 


4 


1.563 


0.321 


0.103 


0.933 


2.192 


4.862** 


Solan 


1 


0.664 


0.365 


0.134 


-0.053 


1.380 


1.816 


Underwood 


1 


-0.027 


0.174 


0.030 


-0.367 


0.314 


-0.153 


Vollands 


6 


0.374 


0.388 


0.150 


-0.386 


1.134 


0.965 


Xin 


6 


0.264 


0.229 


0.052 


-0.184 


0.712 


1.155 


Random 
Effects model 


Total of 
89 effect 
sizes 


0.489 


0.112 


0.013 


0.269 


0.709 


4.360** 



= p<.05, **=p<.01 
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As reported in Table 2, within a random effects model (Lipsey & Wilson, 2002), the weighted 
mean of these 89 corrected effect sizes is 0.49 (sJ=0.74) (z=4.36, p<0.0005). All 89 effect sizes, 
along with the 95 percent confidence intervals are portrayed graphically in Figure 1. The forest 
plot provides a simple visual representation of the amount of information and variation from the 
individual studies that are part of this meta-analysis (with the weighted mean effect size 
appearing as the the right most entry). In the plot, the weighted average (Hedges g) of all effect 
sizes for each study are shown as a vertical line with a diamond plus two tiny rectangles; the 
diamond is weighted effect size and the two small rectangles indicate the limits of the 95 percent 
confidence interval for the effect sizes in any particular study. 



Figure 1. Forest Plot of the Combined Effect Sizes for the 20 Studies 

Meta Analysis of Literacy and Technology in the Middle Grades 



Study Comparison Hedges’s q & 95% Cl Relative weight 



Alfassi 

Dalton 

Gentry 

Fasting 

Hasselbring 

Henao 

Higgins 

Jones 

Kramarski 

Leu 

Ligas 

Liu 

Reinking 88 
Reinking 90 
Rouse 
Salomon 
Solan 

Underwood 

Vollands 

Xin 



Combined 

Read 

Combined 

Combined 

Combined 

Combined 

Combined 

Combined 

Combined 

Combined 

Combined 

Combined 

Combined 

Combined 

Combined 

Combined 

TechnoRead 

ILS 

Combined 

Combined 




Disfavors Techology Favors Technology 



4.17 

5.74 

4.93 

4.88 

5.99 
3.31 
5.13 
5.86 

4.89 
4.68 
6.77 
4.09 
5.23 

3.99 
6.43 

4.48 

4.05 

6.06 
3.84 

5.48 



Random Effects Model 



On the basis of the overall mean weighted effect size, one can and should conclude that the range 
of digital technologies used to ameliorate the reading performance of middle-school students is 
quite effective; in terms of the norms for meta-analysis (Cohen, 1988), this would qualify as a 
“moderate” overall effect size 0.5-0. 8). When examined as percentages, of the 89 effect sizes 
calculated, 26 percent were large (>0.8), 32 percent were moderate (0.5-0. 8), and 42 percent 
were “small” or lower (0.01-0.5). The key term here is range, for there are many types of 
interventions, and clearly some are not any more effective than garden-variety print- oriented 
instruction while others produce sizable advantages over conventional approaches. 
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Examining Simple Effects Within Categories 

Of particular interest for our purposes are a set of very specific comparisons related to the 
variations in programmatic, assessment, and contextual variables. For example, for the 57 effect 
sizes reported for a general, undifferentiated population of middle school students, the mean 
effect size was 0.52, whereas the effect size for targeted populations of students (e.g., students 
classified as possessing learning disabilities or as struggling readers) was 0.32 (N es =29); this 
was a statistically reliable difference, <2=4.42, p<0.05. In comparing meaning focused 
interventions (the combination of mixed, comprehension, vocabulary, and metacognition) with 
those that were code focused (the combination of phonics, phonemic awareness and fluency), we 
found no mean effect size difference favoring one emphasis over another, <2=1.82, p>0.05. The 
mean weighted effect size among studies emphasizing meaning was 0.43 (N es =70) compared to 
0.20 for code (N es =12) 

Study duration, we reasoned, was important, due to the common observation among intervention 
studies that pedagogical experiments often fail to show effects because the intervention does not 
have time to “take hold.” Our results did not confirm the “longer is better” conventional wisdom; 
we instead found a “U-shaped” distribution of effects that, while provocative, was not 
statistically reliable, (7=2.23, p< 0.33: Effect sizes in studies lasting two to four weeks (N es =21, 
ES=0.55) were larger than those in studies lasting less than a week (N es =25, ES=0.48) but much 
larger than those from studies lasting five or more weeks (N es =43, ES=0.34). 

Sample size was a robust predictor of effect size; small n studies (30 or less) produced 14 effect 
sizes averaging 0.77, while large n (31 or more) studies produced 75 effect sizes with a mean of 
0.38, <2=3.24, p<0.20. The possibility exists that the loss of control that comes from larger scale 
implementation of interventions, especially when they are implemented for longer periods of 
time, may result in a loss of power and precision; this is certainly a plausible hypothesis for a 
larger meta-analysis encompassing many other subject areas and target populations. 

Whether a study controlled for pretest equivalency through random assignment (N es =44, 
ES=0.42) or some sort of pretest covariate (N es =45, ES=0.45) did not account for a significant 
amount of variation in effect sizes, <2=.16, p<. 69. Type of test revealed substantial and 
statistically significant differences in effect size, <2=18.62, p<0. 01. Tests produced by test 
companies, largely standardized measures (N es =41, ES=0.30), were less sensitive to treatment 
effects than experimenter-designed assessments (N es =34, ES=0.56). Other (a catchall category) 
tests produced an effect size of 1.05, but there were so few effect sizes (N es =3) that little 
credence can be given to that estimate. 

We also examined effect sizes by their "policy focus," categorizing studies according to whether 
their primary purpose was to (a) reduce the achievement gap, (b) increase technology use in 
general, or (c) improve a specific educational outcome, such as reading comprehension. We 
found no statistically unreliable differences, <2=1.68, p>0. 05. For the 25 effect sizes coming from 
studies designed to improve the achievement gap, the mean effect size was 0.55, where as the 
mean effect size for the studies (N es =30) designed to increase general technology use was 0.36. 
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The mean effect size for the studies (N es =34) designed to improve a specific educational outcome 
as 0.41. 



Another variable of interest is what we dubbed technology source, for lack of a more precise 
label. It contrasts whether the technology originates with a commercial source (e.g., programs 
such as Fast Forward or Accelerated Reader), the researcher’s personal vision of what a 
technological learning environment ought to look like (e.g., Hasselbring & Goin, 2004), or a 
well-studied “delivery system,” such as electronic text with a dictionary available for word 
pronunciation and meaning. When we grouped studies on that variable, the differences were 
quite compelling and statistically significant, <2=32.19, p<0.0001. The 34 effect sizes from the 
commercial studies yielded a mean weighted effect size of 0.28, while the 44 effect sizes from 
delivery system studies averaged 0.34, and the 1 1 effect sizes from researcher-designed 
interventions came in at an effect size of 1.20. There appears to be something special about those 
“tailored” systems designed by individual research teams for specific purposes. 

While it was not central to our investigation, we were interested in whether publication venue 
was a reliable predictor of effect size. So we compared publication in technology journals 
(N es =25, ES=0.54) with literacy journals (n= 30, ES=0.36) with broader educational journals 
(N es =34, ES=0.41). This difference was not statistically reliable, Q=. 1 .73, p>0.05. 

For convenience, these comparisons are summarized below in Table 3. 



Table 3. Summary of Effects Between Levels of Relevant Variables — 

Random Effects Model 



Moderator Variable: 
Levels 


N es 


Mes 

(Hedges’ g) 


Lower 

Confidence 

Interval 


Upper 

Confidence 

Interval 


Q value 3 


Student Sample Size: 






5.216* 


30 or less 


14 


0.772 


0.457 


1.087 




3 1 or more 


75 


0.378 


0.254 


0.501 




Focus of intervention: 






1.828 


Code 


12 


0.200 


-0.107 


0.508 




Meaning 


70 


0.430 


0.302 


0.558 




Type of test: 






18.62** 


Test Co. 


41 


0.30 


0.19 


0.42 




Res Dev 


34 


0.56 


0.21 


0.92 




Other 


3 


1.05 


0.72 


1.38 




Country: 






0.456 


USA 


64 


0.408 


0.274 


0.541 




World 


25 


0.500 


0.267 


0.733 
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Moderator Variable: 
Levels 


N es 


M e s 

(Hedges’ g) 


Lower 

Confidence 

Interval 


Upper 

Confidence 

Interval 


Q value 3 


Duration of Study: 






2.412 


<1 week 


25 


0.481 


0.260 


0.701 




2-4 weeks 


21 


0.545 


0.321 


0.770 




5 weeks + 


43 


0.342 


0.185 


0.500 




Pretest Equivalency: 






0.056 


Rdm Assnt 


44 


0.416 


0.244 


0.589 




Others 


45 


0.445 


0.284 


0.605 




Publication Type: 






1.730 


Tech 


25 


0.541 


0.334 


0.747 




Reading 


30 


0.358 


0.162 


0.554 




Other 


34 


0.400 


0.221 


0.580 




Focus of Policy: 






1.68 


sh Ach Gap 


25 


0.55 


0.34 


0.75 




4 s Techn 


30 


0.36 


0.17 


0.56 




Oth Outc 


34 


0.41 


0.22 


0.59 




Target Population: 






4.42* 


GenEd 


57 


0.52 


0.377 


0.655 




Other 


32 


0.28 


0.100 


0.457 




Tech Source: 






32.19** 


Commercial 


34 


0.28 


0.117 


0.433 




Delivery 


44 


0.34 


0.192 


0.493 




ResDev 


11 


1.20 


0.912 


1.491 




Experimental Design: 






4.432* 


Independent Groups 


59 


0.35 


0.211 


0.482 




Correlated 


30 


0.60 


0.406 


0.802 





a: Q values with p<0.05 indicate that effect sizes differ significantly across levels of the moderator 
variable. *p < 0.05; **p < 0.01 



Summary of Results 

This meta-analysis suggests a number of findings relevant to those interested in the use of 
technology to improve literacy acquisition and instruction at the middle-school level. 

1. As was highlighted by others, very little experimental research exists in this domain. The 
research that does exist focuses mainly on reading comprehension, with a little emphasis 
on metacognitive performances but virtually no attention to issues of motivation and 



Learning Point Associates 



The Effects of Technology on Reading Performance in the Middle-School 

Grades: A Meta- Analysis With Recommendations for Policy — 18 

































engagement. This is all the more surprising given the common claims about the 
motivational value of technology. 

2. A wide range of digital technologies appears to enhance the reading performance of 
middle school students as evidenced by the robust overall effect size obtained in this 
meta- analysis. 

3. A number of specific outcomes merit our attention as a field: 

a. The effect sizes were greater for interventions aimed at general populations than 
those with specific needs (i.e. students who are learning disabled or struggling 
readers). We can only speculate about why this might be the case, and we surely 
need more evidence before reaching a definitive conclusion. However, issues of 
engagement and appropriate levels of support and feedback suggest themselves as 
reasonable explanations. 

b. Standardized measures from test companies were less sensitive to treatment 
effects than researcher-developed measures in several of the studies in this meta- 
analysis. 

c. Studies with smaller sample sizes were much more likely to achieve substantial 
effects than those with larger sample sizes. This counter-intuitive finding is 
puzzling because of what we know about the increase in statistical power that 
comes with larger experimental samples. On the other hand, there may be a trade- 
off between statistical power and experimental precision; that is, it may be easier 
for researchers to maintain a high degree of fidelity to treatment in smaller studies 
because of the greater manageability prospects. 

d. Technologies that were created by a research team had a much larger effect size 
than those technologies either adapted from the commercial market or those that 
merely used the technology as a delivery system. In addition, those created by 
researchers tended to have a clear theoretical focus that was matched by the 
assessments employed by the team. 

e. Studies that used some sort of correlated design (pretests used as covariates for 
posttest or repeated measures designs in which the same subjects cycle through 
different interventions) are more likely to find reliable differences between 
interventions that an independent group designs. 
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Suggestions for Policy and Practice 



We undertook this meta-analysis to determine the state of research-based knowledge about the 
role of technology in improving reading performance in the middle- school grades. Of particular 
importance were digital technologies to improve five areas of literacy acquisition: independent 
strategy use, meta-cognition, reading motivation, reading engagement, and reading 
comprehension. Unfortunately, we were able to locate studies addressing primarily reading 
comprehension and vocabulary, with three studies investigating phonological aspects of reading. 

As we already suggested, this is a grave concern given the hope we collectively express for the 
motivation and engagement that technology ought to promote among learners, particularly 
learners who have not experienced success with conventional curricular tools. That said, the 
research we did locate is encouraging for it shows that these digital learning environments and 
tools can impact learning. These findings have some implications for curricular practice and for 
research. 

Recommendations for Practice 

1. The overall positive impact of technology environments, especially on comprehension 
outcomes, should prompt us to feel comfortable in recommending broader 
implementation of programs that have undergone careful evaluations of their effects on 
student learning. Even though we are tempted to say that educators should consider the 
adoption of programs that possess the same features as those shown to be effective in this 
analysis (e.g., focus on meaning, using a mixed set of technology tools), we think it safer 
for consumers to require careful evaluation of any specific technology program before 
recommending widespread adoption. 

Moreover, the relatively modest impact of commercial programs should prompt us to 
adopt a highly skeptical stance toward claims made by individual vendors and redouble 
our insistence on high quality, independent evaluations of commercial products prior to 
adoption. (In an earlier era, the Educational Products Information Exchange [EPIE] 
served as a kind of Consumer Reports for educational products. With the proliferation of 
software packages and hardware tools, it is needed now more than ever. Perhaps the 
What Works Clearinghouse can serve such a function.) 

2. Program adoption for populations of struggling readers requires even more careful 
evaluation. Our data analyses suggest that positive outcomes for struggling readers are 
much harder to come by. Given the focus of current policy on interventions for struggling 
readers, students with learning disabilities, and other special populations, we believe it 
would be unwise to adopt a program that had not shown an effect for a specific target 
population. We also believe, and explicitly suggest, that much greater emphasis on 
research on tools designed especially for struggling readers is needed. 

3. Current reading assessments, especially commercial assessments, do not appear sensitive 
to the interx’entions possible through technology. Somehow commercial assessments do 
not capture what these interventions are all about, and we believe, as we suggest below in 
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our recommendations for future research, that we need more assessment instruments that 
exhibit greater instructional sensitivity. The current crop of standardized tests is held to a 
high standard for criteria of reliability and concurrent validity, but there is little evidence 
of their instructional validity (i.e., sensitivity to changes in performance due to 
instruction). We need assessment instruments that provide more sensitive tests of the 
efficacy of instructional interventions in this burgeoning technology enterprise. 

Recommendations for Future Research 

As we consider future research in this area, we believe that the following recommendations 

deserve our collective consideration and action: 

1. The present data reinforce the many existing calls for more research in this area. If one 
puts together three key findings from this meta-analysis [(a) there are not enough 
experimental studies, (b) there is a narrow focus on cognitive outcomes (comprehension), 
and (c) the existing studies show promising effects on literacy acquisition], one is led to 
the conclusion that we should continue, perhaps even expand, funding for research on 
technological interventions to improve literacy acquisition at the middle-school level. As 
promising as it is, there is just too little research to allow for us to make strong claims 
about the efficacy of technology on literacy. 

After multiple filtering phases to ensure the correct population, the appropriate 
intervention, and rigorous research, only 20 studies survived. As such, only one of the 
five initial research questions could be answered. Funding for future research should 
move beyond existence proofs (technology can make a difference) to provide more 
specific and nuanced information about when, where, why and how technology can 
support teaching and learning for middle school literacy acquisition. Our call for research 
echoes the concerns and needed directions expressed by leading authors in literacy and 
technology (e.g., Labbo & Reinking, 1999; Leu, in press). 

2. Future research may need to balance issues of focus against standards of control and 
precision. The largest effect sizes in this meta-analysis were from studies used a smaller 
n; moreover, there is a tendency, albeit nonsignificant for shorter studies to produce 
greater effects. This suggests that research studies that last too long might be open to 
maturation effects or other confounding variables. Research that takes place too quickly 
might not provide time for the intervention to take hold. Studies with large sample sizes 
might compromise researcher control that would be available in a smaller, more 
manageable study. 

The larger issue implied by this recommendation is the question of what research 
methods ought to be employed in the conduct of research in this or any other educational 
arena. Complementarity, it seems to us, is called for in this arena. The complementarity 
principle would suggest that in any venture, we begin with small-scale descriptive studies 
before moving on to more careful design or formative experiments that help us narrow 
the range of relevant variables in anticipation of carefully controlled randomized 
experiments and, finally, studies of what happens in the scaling up process. This principle 
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seems even more important in a relatively new field, such as the development of digital 
tools to enhance literacy learning. 

3. As' afield, we should develop a master codebook that could seri’e the research community 
as a heuristic for analyzing digitcd technologies and their impact on literacy acquisition 
in the middle-school grades (perhaps beyond ). This recommendation originated in the 
work of Cavanaugh et al. (2004), and its utility was once again demonstrated in this 
meta- analysis. There are many complexities related to studying digital tools and their 
impact on literacy acquisition; a research field without such a heuristic will find it 
difficult to compare outcomes or to come to any concrete conclusions about implications 
for teaching literacy at this level. We found the Waxman system (see Table 1 and 
Appendix E), with a few tweaks to make it more literacy-centric, to be quite useful to us, 
and we would recommend it to others. 

4. Future research needs to examine the relation between commercial products and 
researcher-developed technology interx’entions. Little research has investigated 
commercial technology products used for improving literacy acquisition at the middle- 
school level. However, this meta-analysis has provided evidence that researcher- 
developed technologies seem to be more effective than their commercial counterparts. 
This finding could be due to the fact that in several of the studies in our corpus the 
researchers also developed the measures used to determine the effectiveness of the 
program. As such, the measures may be “testing the tool” rather than striving for transfer. 

It is also possible that by working together researchers, educators, and technologists are 
better able to create a system tailored specifically to meet the needs of particular 
audiences than commercial products trying to serve large audiences. As we will argue 
later, stronger, more valid and more reliable measures (along with a better coding 
heuristic) will help address this issue of whether the difference between commercial 
programs is an artifact of the match between assessments and programs or a result of 
more careful implementation of a learning environment. If it is the case that researchers 
are better able to develop effective literacy tools, then better dissemination plans need to 
be enacted to share these benefits with practitioners — and possibly with the educational 
publishing community, so that they can infuse promising new technological innovations 
into their products. 

Two other recommendations for research are only indirectly implied — and certainly not 
licensed — by our meta-analysis, but both are worth mentioning because they are so central to 
the future of research in this area. 

1. Assessment. Our meta-analysis did unearth assessment, especially the question of 
what sorts of assessments should count as evidence of the efficacy of a technological 
intervention, as an issue. We believe that the research on digital tools for middle 
school literacy acquisition should include a focus on developing measures to evaluate 
outcomes that are generalizable, comparable, and replicable. 

We found that researcher-developed measures yielded greater effect sizes than 
external standardized tests. Is it because these highly curricular-embedded, 
researcher-developed tests are more relevant to the treatment and hence more valid — 
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or just a reflection of what happens when a program teaches to the test and in the 
process compromises the generalizability of the intervention, compared to what might 
have been achieved with standardized measures? We are not sure, but we are sure that 
the issue needs our scholarly attention. 

2. Engaging teachers in technology interx’entions. Few would argue with the assertion 
that teachers need practical information to learn how to best use digital tools in the 
classroom. As a research field, we are still a long way from helping teachers 
implement effective classroom technology systems. Thus, we would welcome 
research on how to assist teachers in implementing technology. However, more is 
needed. Most of the interventions in our analysis put the researcher at the center of 
the classroom implementation of the technology and positioned the teacher as a 
bystander. 

We would like to see collaborative research that engages teachers from the outset in 
the design and implementation of classroom digital tools. Only when researchers 
engage teachers from the conceptualization of their technology tools can researchers 
benefit from the wisdom of teaching in their designs. Only when researchers expand 
their methodological repertoire to include iterative design experiments in advance of 
randomized field trials will there be a place for teachers to engage in full and 
continuous collaboration. 

Finally, a comment for those who would raise the issue of whether it is worth doing a meta- 
analysis on a corpus of only 20 studies: There appears to be a general belief among some 
educational researchers that a large number of studies must be included in a meta-analysis 
project in order to draw substantive conclusions. For example, the National Reading Panel on 
Technology (NRP, 2000) decided not do a meta- analysis because there were only 21 studies 
identified. Given the wider range of grades and questions asked in that initiative, perhaps the 
number of studies would not have been sufficient. However, even in that effort, we should note 
that even though no meta- analysis was carried out, the NRP found that all 21 studies indicated 
the positive effects of technology on reading performance and reached positive conclusions 
about its efficacy. 

By focusing on an undefined and statistically unsupported assumption about a minimum number 
of studies to carry out a meta-analysis the more relevant concept of heterogeneity is obscured. 
Heterogeneity refers to the fact that studies grouped together in a systematic review will differ in 
a variety of systematic and random ways. The differences can be in experimental design, 
outcomes measures reported, and other factors. Statistically, heterogeneity means that observed 
treatment effects differ more from each other than one would expect from random factors alone. 
Thus the more important task to carry out in meta-analysis is to more precisely abstract useful 
and homogeneous information from the studies and manipulations of the specific construct(s) of 
interest. 

Hardy and Thompson (1998) examined various factors that impact the power of a heterogeneity 
test. They included such factors as the number of studies, the total information available, and the 
distribution of weights. Their findings show that the power increases with the total amount of 
information not merely the numbers of studies in a meta-analysis. Hardy and Thompson also 
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showed that if a particular study contributes an inordinately large amount to the overall weighted 
mean effect size, the power is substantially lowered. 

Given two important facts — (1) that this meta- analysis had a very specific focus of (reading and 
technology in Grades 6-8) and (2) that our admittedly small number of studies provided a large 
amount of information about the effects of technology on reading — we can be more confident 
about our findings and conclusions. Granted, our meta- analysis would be stronger if there were 
many more experiments available, but we believe we have made a solid beginning in looking at 
technology and reading. 

We also note that no single study in our meta-analysis overwhelmed the other studies in terms of 
contributions to the overall weighted mean (this can be seen by examining the column of relative 
weight on the forest plot in Figure 1). Hardy and Thompson (1998) conclude their article by 
pointing out that that expert judgment deserves as much weight as statistical analyses of 
heterogeneity in determining weight and significance. 

Our confidence in recommending more policy and research attention to technology, thankfully, 
is supported by the dual criteria of statistical scrutiny and wisdom. We believe the time has come 
to take technology more seriously as a component of middle-school literacy curriculum and 
pedagogy. 
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Appendix A 

Keywords Used for Web Searches 



adolescent 

achievement 

cognitive 

computer 

computer-based instruction 

comprehension 

digital media 

educational technology 

electronic media 

evaluation 

experiment 

hypermedia 

hypertext 

instruction 

Internet 

intervention 

language 

learning 

learning environment 
metacognition 
middle school 
middle grades 
multimedia 
online 

open learning 
quantitative 
quasi-experimental 
phonemic awareness 
pretest, posttest 
print 

randomized 

reading 

research 

strategy 

technology 

textbook 

validity 

vocabulary 

web-based 

6th grade (or sixth grade) 
7th grade (or seventh grade) 
8th grade (or eighth grade) 
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Appendix B 

Academic and Educational Databases 

Blackwell Science Synergy 
Directory of Open Access Journals 
Ebsco Research Databases 
ERIC 

Gale Group Databases 
JSTOR 
Ingenta Select 
Kluwer 

Lawrence Erlbaum Journals 
MetaPress 

OCLC FirstSearch - Periodical Abstracts 
Ovid 

ProQuest Education 

Psychlnfo 

PubMed 

Sage Publications 
SpringerLink 
Wiley Interscience 
Wilson Education. 
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Appendix C 

Educational Technology and Reading Journals 



American Educational Research Journal 
American Journal of Distance Education 
American Annals of the Deaf 

Association for the Advancement of Computing in Education 

Behavior Research Methods, Computers and Instrumentation 

Children’s Literature in Education 

Communication Disorders Quarterly 

Computer Science Education 

Computers in Human Behavior 

Computers in the School 

Computers & Education 

Contemporary Educational Psychology 

Disability and Rehabilitation 

Distance Education 

Economics of Education Review 

Education and Information Technologies 

Education Policy Analysis Archives 

Educational Psychology 

Educational Psychologist 

Educational Technology & Society 

Educational Technology Research and Development 

Electronic Journal for the Integration of Technology in Education 

Elementary School Journal 

E-learning 

E-Leaming and Education 
Human and Computer Interaction 
Human Factors 

Information Technology, Learning and Performance 

Interactive Learning Environments 

Journal of Asynchronous Learning Networks 

Journal of Adolescent and Adult Literacy 

Journal of Applied Psychology 

Journal of Computer Assisted Learning 

Journal of Computer Mediated Communication 

Journal of Computers in Math and Science Teaching 

Journal of Computing in Childhood Education 

Journal of Distance Education 

Journal of Distance Learning 

Journal of Education Technology Systems 

Journal of Educational Computing Research 

Journal of Educational Computing, Design & Telecommunications 

Journal of Educational Media 

Journal of Educational Psychology 
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Journal of Educational Research 

Journal of Educational Technology Research and Development 

Journal of Experimental Child Psychology 

Journal of Experimental Psychology 

Journal of Information Technology Education 

Journal of Interactive Media in Education 

Journal of Interactive Learning Research 

Journal of Interactive Online Learning 

Journal of Learning Disabilities 

Journal of Literacy Research 

Journal of Reading Behavior 

Journal of Research in English 

Journal of Research on Technology in Education 

Journal of Teaching, Learning and Assessment 

Journal of Technology Education 

Journal of Technology Studies 

Journal of the Learning Sciences 

Language, Learning, and Technology 

Language and Leading with Technology 

Learning Disabilities: Research and Practice 

Learning and Instruction 

Learning and Leading with Technology 

Journal of Special Education Technology 

Open Education 

Reading Online 

Reading Psychology 

Reading Research and Instruction 

Reading Research Quarterly 

Reading and Writing 

Reading and Writing Quarterly 

Research in Education Education 

Scientific Studies of Reading 

Technology and Learning 

TechKnowLogia: International Journal of Technologies for the Advancement of 
Knowledge and Learning 
The Reading Matrix. 
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Appendix D 
International Journals 



Australian Educational Computing 
Australian Journal of Education 

Australian Journal of Educational and Developmental Psychology 

Australian Journal of Educational Technology 

Australian Journal of Language and Literacy 

Australasian Journal of Educational Technology 

British Educational Research Journal 

British Journal of Educational Psychology 

British Journal of Educational Technology 

British Journal of Learning Disabilities 

British Journal of Special Needs Education 

Canadian Journal of Education 

Canadian Journal of Educational Communication 

Canadian Journal of Experimental Psychology 

Canadian Journal of Learning and Technology 

Ensenanza de las Ciencias 

European Education 

European Journal of Cognitive Psychology 
European Journal of Education 
European Journal of Psychology of Education 
European Journal of Special Needs Education 
Infancia y Aprendizaje 

International Journal of Artificial Intelligence in Education 
International Journal of Educational Technology 
International Journal on E-learning 

International Review of Research in Open and Distance Learning 

Language & Literacy: A Canadian Educational e-joumal 

Oxford Review of Education 

Revista de Ciencias Humanas 

Revista Electronica de Investigacion Educativa 

Revista Iberoamericana de Educacion 

Turkish Online Journal of Distance Education 

Scandinavian Journal of Educational Research 

Scandinavian Journal of Psychology 
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Appendix E 

Meta- Analysis Coding for Reading and Technology Studies 
(Revised from Waxman, 2003) 

1. Study Characteristics 

a. Author (Report last name, first; e.g., Doe, John). 

b. Year of Study (Report year of study; e.g., 2000). 

c. Number of Comparisons Within Study (Report number; e.g., 1 or 2 or 3). 

d. Student Sex (Males = 1; Females = 2; Mixed or not specified = 3). 

e. Grade Level (Unspecified = 0; on target (6-8th grade)=l; on target + below=2; on 
target + above=3; on target plus above and below=4; Just below=5; just above=6; 
other=9) 

f. Unit of Analysis (Unspecified = 0; Individual = 1; Class = 2; School = 3; District 
= 4; State = 5; Mixed = 6). 

g. Student Sample Size (Report actual sample size of both groups; e.g., 4,024). 

h. School Sample Size (Report actual sample size; e.g., 4,024). 

i. Publication Features (Technology journal = 1; Reading or Literacy journal = 2; 
Special Education=3, Other educational journal = 4; not in archival literature = 5). 

j. Target Population (Unspecified=0; General Ed=l; Special Education=2; 
Struggling Readers=3; Language ESL in USA=4; Language ESL in Other 
Country=5; Other second language learning=6; Other=9). 

k. Students’ Ethnicity (Unspecified = 0; Black = 2; Hispanic = 3; Asian = 4; White = 
5; Mixed = 6; Other = 9). 

l. Students’ Socioeconomic Status (Unspecified = 0; Lower = 1; Lower middle = 2; 
Middle = 3; Upper middle = 4; Upper = 5; Mixed = 6). 

m. Country (Unspecified = 0; USA = 1; Canada = 2; Mexico/Latin America = 3; 
Europe = 4; Asia = 5: South America = 6; Cross-Cultural = 7; Middle East = 8; 
Other=9) 

n. Geographical Region (if in USA) (Northeast = 1; Southeast = 2; Midwest = 3; 
South Central = 4; Southwest = 5; Northwest = 6; Mixed = 7; Other = 9). 

o. School Type (Unspecified = 0; Public = 1; Private = 2; Special school = 3; Mixed 
= 4; Other = 9). 

p. Community Type (Unspecified = 0; Urban = 1; Rural = 2; Suburban = 3; Mixed = 
4; Other = 9). 

q. Content Area (Content area where reading technology is used. Unspecified = 0; 
Technology = 1; Reading = 2; Mathematics = 3; Social studies = 4; Science = 5; 
Reading and math = 6; Language arts = 7; Foreign language = 8; Mixed = 9; 

Other =10). 

2. Quality of Study Indicators 

a. Method of Observation of Independent Variable (i.e., how was the technology 
intervention documented — how did they look to see if the kids did it). 

Unspecified = 0; Systematic observation = 1; Informal observation = 2; Student 
survey or interview = 3; Teacher survey or interview = 3; Administrator survey or 
interview = 4; Computer logs = 5; Multiple methods = 6; Other = 9). 



Learning Point Associates 



The Effects of Technology on Reading Performance in the Middle-School 

Grades: A Meta- Analysis With Recommendations for Policy — 35 




b. Pretest Equivalency (Has the initial differences between the two groups been 
accounted for? Unspecified = 0; Statistical Control (e.g., ANCOVA, regression) = 
1; Random Assignment = 2; Statistical Control and Random Assignment = 3; 

Gain Scores = 4; Matching = 5; Other = 9). 

c. Name of test or measure (report actual name) 

d. Reported Reliability of Measures (Unspecified = 00; Actual reliability statistic 
(e.g., 70 or 83). 

e. Manner in Which Outcome Scores Are Reported (Unspecified = 0; Standard 
scores = 1; Raw scores = 2; Percentile ranks = 3; Gain scores = 4; Residualized 
Posttest scores = 5; Other=9). 

f. Duration of Study (Unspecified = 00; List the number of weeks that the 
implementation of the technology occurred). 

g. Cognitive Outcomes (Unspecified = 0; Testing company standardized 
achievement test = 1; Federal/national standardized test = 2; State-level 
achievement test = 3; District-level achievement test = 4; School-level test = 5; 
Teacher-made test = 6; Researcher-developed test = 7; Test developed by other 
researchers=8; Authentic assessment = 9; Creativity test = 10; Higher-level 
thinking test =11; Other = 19). 

h. Affective Outcomes (Unspecified = 0; Student attitudes toward computers = 1; 
student attitudes toward reading = 2; student attitudes other = 3; Academic self- 
concept or motivation = 4; student preference for electronic media = 5; Other = 
19). ' 

i. Behavioral Outcomes (Unspecified = 0; Student time-on-task = 1; student 
perseverance = 2; Tasks attempted = 3; Tasks completed = 3; Success rate = 4; 
Positive peer interaction = 5; Interactivity with computers = 6; Other = 9). 

j. Effect Size Coefficient (actual coefficient) 

k. Statistics (Statistics used in determining effect size; Means = 1; t-value = 2; F- 
value = 3; Chi-square = 4; Mixed = 5; Anova=6; Other=9). 

l. Weight (One divided by the actual number of comparisons in the study, e.g., 3 
comparisons = 1/3 or .333). 

3. Sources of Invalidity 

a. Type of Design (Quasi-experimental/nonrandomized one group pretest-posttest = 
1; Nonrandomized static-group comparison = 2; Nonrandomized pre-post control 
group = 3; Time series = 4; Randomized posttest-only control group = 5; 
Randomized pre-post control group = 6; Repeated Measures = 7; Other=9). 

b. History (Have specific events occurred between the first and second measurement 
in addition to the experimental variable? Adequately controlled by design = 1 ; 
Definite weakness of design = 2; Possible source of concern = 3; Not a relevant 
factor = 4). 

c. Maturation (Are there processes within the participants operating as a function of 
the passage of time [e.g., growing older, more tired] that might account for 
changes in the dependent measure? Adequately controlled by design = 1 ; Definite 
weakness of design = 2; Possible source of concern = 3; Not a relevant factor = 

4). 
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d. Testing (Is there an effect of taking a test upon the scores of a second testing? 
Adequately controlled by design = 1; Definite weakness of design = 2; Possible 
source of concern = 3; Not a relevant factor = 4). 

e. Instrumentation (Do changes in calibration or observers’ scores produce changes 
in the obtained measurement? Adequately controlled by design = 1; Definite 
weakness of design = 2; Possible source of concern = 3; Not a relevant factor = 

4). 

f. Statistical Regression (Have groups been selected on the basis of their extreme 
scores? Adequately controlled by design = 1; Definite weakness of design = 2; 
Possible source of concern = 3; Not a relevant factor = 4). 

g. Selection Bias (Have biases resulted in the differential selection of comparison 
groups? Adequately controlled by design = 1; Definite weakness of design = 2; 
Possible source of concern = 3; Not a relevant factor = 4). 

h. Mortality (Has there been a differential loss of participants from the experimental 
and control groups? Adequately controlled by design = 1; Definite weakness of 
design = 2; Possible source of concern = 3; Not a relevant factor = 4). 

i. Selection-Maturation Interaction (Is there an interaction between extraneous 
factors such as history, maturation, or testing and the specific selection 
differences that distinguish the experimental and control groups? Adequately 
controlled by design = 1; Definite weakness of design = 2; Possible source of 
concern = 3; Not a relevant factor = 4). 

j. Reactive or Interaction Effect of Testing (Does pretesting influence the 
participants’ responsiveness to the experimental variable, making the results for a 
pretested population unrepresentative of the effects of the experimental variable 
for the unpretested universe from which the participants were selected? 
Adequately controlled by design = 1; Definite weakness of design = 2; Possible 
source of concern = 3; Not a relevant factor = 4). 

k. Interaction of Selection Biases and Treatment (Are there selective factors upon 
which sampling was based which interact differentially with the experimental 
variable? Adequately controlled by design = 1; Definite weakness of design = 2; 
Possible source of concern = 3; Not a relevant factor = 4). 

l. Reactive Effects of Experimental Arrangements (Are there effects of the 
experimental setting that would preclude generalizing about the effect of the 
experimental variable upon persons being exposed to it in nonexperimental 
settings? Adequately controlled by design = 1; Definite weakness of design = 2; 
Possible source of concern = 3; Not a relevant factor = 4). 

m. Multiple-Treatment Interference (Are there nonerasable effects of previous 
treatments applied to the same participants? Adequately controlled by design = 1; 
Definite weakness of design = 2; Possible source of concern = 3; Not a relevant 
factor = 4). 

n. Statistical Power (Is the sample size large enough to reject the null hypothesis at a 
given level of probability, or are the estimate coefficients within reasonably small 
margins of error? ; i.e. each group has how many) [a sample > 60 for groups such 
as classes, schools, or districts; a sample >100 for individuals]. Probable threat [< 
60 for groups or < 100 for individuals as the unit of analysis] = 1; Adequately 
minimized [> 60 for groups; >100 for individuals] = 2). 



Learning Point Associates 



The Effects of Technology on Reading Performance in the Middle-School 

Grades: A Meta- Analysis With Recommendations for Policy — 37 




4. Reading Characteristics 

a. The Focus of the intervention (what they did) 

• 1 = Phonics 

• 2 = Phonemic awareness 

• 3 = Vocabulary 

• 4 = Reading comprehension 

• 5 = Fluency 

• 6 = independent reading 

• 7 = Mixed 

• 8 = Meta-cognition 

• 9 = Content Learning 

• 19 = Other 

b. The focus of the outcome measures (what they observed) 

• 1 = phonics 

• 2 = phonemic awareness 

• 3 = vocabulary 

• 4 = reading comprehension 

• 5 = fluency 

• 6 = reading volume 

• 7 = reader response (response to literature) 

• 8 = Spelling 

• 9 = Word Recognition 

• 10 = Aggregate 

• 11 = Meta-cognition 

• 12 = Motivation 

• 19 = Other 

5. Technology Characteristics 

a. Type of Technology (Unspecified = 0; PCs = 1; Laptops = 2; Networked labs = 3; 
HP calculators = 4; Multimedia = 5; Other = 9). 

b. Software (Unspecified = 0; Tutorial = 1; Drill-and-practice = 2; Exploratory 
environment [e.g., simulations, microworlds, hypermedia, and hypertext] = 3; 
Tools for other tasks [e.g., word processor for writing, e-mail, or computer- 
conference for course assignments] = 4; Programming language = 5; Computer- 
supported Independent Reading Environment=6; Other = 9). 

c. Technology Resources/Support Available (Unspecified = 0; No resources = 1; 
Minimal resources = 2; Adequate resources = 3; Ample resources = 4; Other = 9). 

d. Role/Focus of Technology (Unspecified = 0; Productivity = 1; Delivery system 
[e.g., ILS] = 2; Resource [e.g., Internet] = 3; Other = 9). 

e. Quantity of Technology (Unspecified = 0; Few [< 3 per classroom] = 1; Average 
[4-8 per classroom] = 2; Ample [> 9 per classroom] = 3; Other = 9). 

f. Number of Computer Sessions (Unspecified = 0; List number of sessions [e.g., 
12]). 

g. Duration of Computer Sessions (Unspecified = 0; List number of average minutes 
per sessions [e.g., 40]). 
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h. Teachers’ Experience with Technology (Unspecified = 0; None = 1; Minimal 
experience = 2; Average = 3; Experienced = 4; Very experienced = 5). 

i. Students’ Experience with Technology (Unspecified = 0; None = 1; Minimal 
experience = 2; Average = 3; Experienced = 4; Very experienced = 5). 

j. Teacher Training in Technology (Unspecified = 0; List hours of training (e.g., 

15). 

k. Feedback and Assessment Practices (Unspecified = 0; No feedback = 1; Minimal 
feedback = 2; Elaborate feedback = 3; Other = 9). 

l. Learning Responsibility (Unspecified = 0; Student controlled = 1; Teacher 
directed = 2; System directed = 3; Mixed = 4; Other = 9). 

m. Task Difficulty (Unspecified = 0; Difficult = 1; Moderately difficult = 2; Not 
difficult = 3; Mixed levels of difficulty = 4; Other = 5). 

n. Type of Learning Task (Unspecified = 0; Basic skills/factual learning = 1; 
Problem solving = 2; Inquiry /investigation = 3; Project-based = 4; Mixed types = 
5; Other = 9). 

o. Type of Technology Program (Unspecified = 0; Basic skills/factual learning = 1; 
Problem solving = 2; Inquiry = 3; Mixed types = 4; Other = 9). 

p. Pattern of Student Computer Use (Unspecified = 0; Teacher use only = 1; 
Presentation station = 2; One student per computer = 3; Two students per 
computer = 4; 3-5 students per computer = 4; > 5 students per computer = 6; 
Mixed pattern = 7; Other = 9). 

q. Percentage of Students Using Computers (Unspecified = 0; > 10 percent = 1; 10- 
25 percent = 2; 26-50 percent = 3; 51-75 percent = 4; 76-90 percent = 5; > 90 
percent = 6). 

r. Objectives of Computer Use (Unspecified = 0; Remediation of skills not learned 
= 1; Expressing themselves in writing = 2; Communicating electronically with 
other people = 3; Finding out about ideas and information = 4; Analyzing 
information = 5; Presenting information to an audience = 6; Improving computer 
skills = 7; Learning to work collaboratively = 8; Learning to work independently 
= 9; Multiple Objectives = 10; Other = 19). 

s. AEGIS of Technology (Unspecified=0; commercial venture (e.g., Fast Forward or 
Acc Reader)=l; experimenter developed=2; externally (feds or foundation) 
funded=3; 9=other) 

6. Instructional/Teaching Characteristics 

a. Joint Productive Activity/Collaboration (e.g.. Designs instructional activities 
requiring student collaboration to accomplish a joint product; monitors and 
supports students collaboration in positive ways. Unspecified = 0; No evidence = 
1; Some evidence = 2; Extensive evidence = 3). 

b. Challenging Activities (e.g., Designs instructional tasks that advance students’ 
understanding to more complex levels. Assures that students — for each 
instructional topic — see the whole picture as a basis for understanding the parts. 
Unspecified = 0; No evidence = 1; Some evidence = 2; Extensive evidence = 3). 

c. Instructional Conversation (e,g, Arranges the classroom to accommodate 
conversational between the teacher and a small group of students on a regular and 
frequent basis. Guides conversation to include students’ views, judgments, and 
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rationales using text evidence and other substantive support. Unspecified = 0; No 
evidence = 1; Some evidence = 2; Extensive evidence = 3). 

d. Setting (Unspecified = 0; Classroom = 1; Networked lab within class = 2; 
Computer lab in school = 3; Other = 9). 

e. Mode of Instruction (Unspecified = 0; Whole-group instruction = 1; Paired = 2; 
Small-group instruction [3-5 members] = 3; Individualized = 4; Mixed = 5; Other 
= 9). 

f. Role of Teacher (Unspecified = 0; Deliverer of knowledge = 1; Facilitator of 
groups/student learning = 2; Modeling processes [e.g., problem solving] = 3; 
Mixed = 4; Observer=5; Other = 9). 

g. Teacher Qualifications (Unspecified = 0; Alternatively certified or provisional 
certificate = 1; Certified in content area = 2; Not certified in content area = 3; 
Other = 9). 

7. Policy 

a. Level (Unspecified = 0; School = 1; District = 2; State = 3; Federal = 4; Other = 
5). 

b. Focus (Unspecified = 0; Reducing achievement gaps = 1; Increased use of 
technology = 2; Increased specific type of use = 3; Improve Specific Educational 
Outcomes=4; Other = 9). 
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Appendix F 

Statistics for the 89 Effect Sizes in the Analysis 



Study name 


Comparison 


Hedges's g 


Standard 

error 


Variance 


Lower 

limit 


Upper 

limit 


Z- 

Value* 


Alfassi 


Accuracy 


-0.237 


0.327 


0.107 


-0.878 


0.405 


-0.723 


Alfassi 


ReadComp 


1.674 


0.381 


0.145 


0.928 


2.421 


4.395 


Alfassi 


ReadRate 


1.009 


0.347 


0.120 


0.329 


1.688 


2.907 


Dalton 


Read 


0.424 


0.204 


0.042 


0.023 


0.825 


2.075 


Gentry 


Digital 


0.135 


0.279 


0.078 


-0.411 


0.681 


0.484 


Gentry 


PictAbs Digital 


0.342 


0.280 


0.079 


-0.207 


0.892 


1.221 


Gentry 


PictPres Digital 


-0.072 


0.278 


0.078 


-0.618 


0.473 


-0.260 


Fasting 


OS400 


1.013 


0.291 


0.084 


0.443 


1.583 


3.485 


Fasting 


OS400Follow 


0.034 


0.273 


0.075 


-0.502 


0.569 


0.123 


Fasting 


SL60 


0.917 


0.288 


0.083 


0.353 


1.480 


3.188 


Fasting 


SL60Follow 


0.452 


0.277 


0.077 


-0.090 


0.995 


1.634 


Fasting 


Spelling 


1.244 


0.299 


0.090 


0.658 


1.830 


4.159 


Fasting 


SpellingFollow 


-0.155 


0.274 


0.075 


-0.691 


0.382 


-0.565 


Hasselbring 


Outcome 1 


0.653 


0.183 


0.033 


0.295 


1.010 


3.575 


Hasselbring 


Outcome2 


0.645 


0.182 


0.033 


0.288 


1.003 


3.538 


Hasselbring 


Outcome3 


0.379 


0.179 


0.032 


0.027 


0.730 


2.111 


Hasselbring 


Outcome4 


0.405 


0.180 


0.032 


0.053 


0.757 


2.257 


Henao 


DetGood 


0.000 


0.428 


0.183 


-0.839 


0.839 


0.000 


Henao 


DetPoor 


0.285 


0.431 


0.185 


-0.559 


1.129 


0.661 


Henao 


ImpGood 


0.872 


0.450 


0.202 


-0.010 


1.754 


1.938 


Henao 


ImpPoor 


1.517 


0.491 


0.241 


0.555 


2.479 


3.090 


Higgins 


Grade 


0.514 


0.259 


0.067 


0.006 


1.022 


1.984 


Higgins 


Raw 


0.686 


0.262 


0.069 


0.171 


1.200 


2.613 


Jones 


LanExp 


0.332 


0.193 


0.037 


-0.046 


0.710 


1.721 


Jones 


LanExp Sessions 


0.357 


0.193 


0.037 


-0.021 


0.735 


1.849 


Jones 


LanMech 


0.201 


0.192 


0.037 


-0.175 


0.578 


1.048 


Jones 


LanMech Sessions 


0.441 


0.194 


0.037 


0.062 


0.821 


2.280 


Jones 


Rcomp 


0.413 


0.193 


0.037 


0.034 


0.791 


2.134 


Jones 


Rcomp Sessions 


0.140 


0.192 


0.037 


-0.236 


0.516 


0.729 


Jones 


RVocab 


0.532 


0.194 


0.038 


0.151 


0.912 


2.735 


Jones 


RVocab Sessions 


0.130 


0.192 


0.037 


-0.246 


0.507 


0.679 


Jones 


Spelling 


0.265 


0.192 


0.037 


-0.112 


0.642 


1.376 


Jones 


Spelling Sessions 


0.530 


0.194 


0.038 


0.149 


0.910 


2.726 


Kramarski 


Metecog 


-1.057 


0.292 


0.085 


-1.630 


-0.485 


-3.618 


Kramarski 


Motiv 


0.704 


0.282 


0.079 


0.152 


1.256 


2.499 


Kramarski 


Rcomp 


-0.259 


0.274 


0.075 


-0.797 


0.278 


-0.945 


Leu 


Control High 


0.836 


0.303 


0.092 


0.242 


1.429 


2.759 


Leu 


Control Lo 


0.943 


0.313 


0.098 


0.330 


1.555 


3.014 


Leu 


Control Med 


0.888 


0.314 


0.099 


0.272 


1.504 


2.824 


Leu 


DrpCtrl High 


0.223 


0.291 


0.085 


-0.348 


0.793 


0.765 


Leu 


DrpCtrl Lo 


0.016 


0.296 


0.088 


-0.564 


0.596 


0.054 


Leu 


DrpCtrl Med 


0.116 


0.300 


0.090 


-0.471 


0.704 


0.388 


Ligas 


6:95-96 


-0.143 


0.060 


0.004 


-0.261 


-0.024 


-2.363 


Ligas 


6:96-97 


-0.084 


0.060 


0.004 


-0.202 


0.035 


-1.388 


Ligas 


6:97-98 


-0.174 


0.060 


0.004 


-0.293 


-0.056 


-2.884 


Ligas 


6:98-99 


-0.153 


0.060 


0.004 


-0.271 


-0.034 


-2.531 
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Study name 


Comparison 


Hedges's g 


Standard 

error 


Variance 


Lower 

limit 


Upper 

limit 


Z- 

Value* 


Ligas 


Hrsl2+5- 


0.698 


0.168 


0.028 


0.368 


1.028 


4.147 


Liu 


LS Science 


1.676 


0.416 


0.173 


0.861 


2.492 


4.029 


Liu 


RegEdScience 


3.047 


0.194 


0.038 


2.666 


3.428 


15.675 


Liu 


Tag Science 


3.314 


0.425 


0.180 


2.482 


4.146 


7.806 


Reinking 88 


Comp All 


1.011 


0.259 


0.067 


0.504 


1.518 


3.908 


Reinking 88 


CompSE 


0.754 


0.252 


0.064 


0.261 


1.248 


2.994 


Reinking 88 


CompText 


-0.108 


0.243 


0.059 


-0.586 


0.369 


-0.445 


Reinking 88 


LEALL 


-0.247 


0.244 


0.060 


-0.725 


0.232 


-1.010 


Reinking 88 


LESE 


-0.231 


0.244 


0.060 


-0.709 


0.248 


-0.944 


Reinking 88 


LETE 


-0.368 


0.245 


0.060 


-0.849 


0.113 


-1.498 


Reinking 88 


PRALL 


-0.351 


0.245 


0.060 


-0.831 


0.130 


-1.430 


Reinking 88 


PRSE 


-0.189 


0.244 


0.059 


-0.667 


0.289 


-0.774 


Reinking 88 


PRTE 


-0.349 


0.245 


0.060 


-0.829 


0.132 


-1.422 


Reinking 88 


Rail 


1.831 


0.291 


0.085 


1.261 


2.401 


6.297 


Reinking 88 


Rase 


0.745 


0.252 


0.063 


0.251 


1.238 


2.959 


Reinking 88 


Rate 


0.067 


0.243 


0.059 


-0.410 


0.544 


0.274 


Reinking 90 


Compall 


0.631 


0.364 


0.133 


-0.084 


1.345 


1.731 


Reinking 90 


Compglo 


0.354 


0.358 


0.128 


-0.348 


1.056 


0.989 


Reinking 90 


Compsel 


0.362 


0.358 


0.128 


-0.340 


1.064 


1.010 


Reinking 90 


Defs 


0.833 


0.371 


0.138 


0.105 


1.561 


2.244 


Reinking 90 


Vail 


1.452 


0.402 


0.161 


0.665 


2.240 


3.615 


Reinking 90 


Vglo 


0.044 


0.355 


0.126 


-0.652 


0.741 


0.124 


Reinking 90 


Vsel 


1.163 


0.386 


0.149 


0.407 


1.919 


3.016 


Rouse 


CompReadEdge 


0.057 


0.093 


0.009 


-0.126 


0.239 


0.609 


Rouse 


Overal CELF 


0.062 


0.214 


0.046 


-0.357 


0.481 


0.289 


Rouse 


SFA Assess 


0.059 


0.104 


0.011 


-0.144 


0.262 


0.566 


Rouse 


State Read Test 


0.063 


0.094 


0.009 


-0.121 


0.247 


0.675 


Salomon 


CntExpComp 


1.395 


0.311 


0.097 


0.785 


2.005 


4.480 


Salomon 


CrtlExpComp 


1.730 


0.331 


0.110 


1.081 


2.379 


5.226 


Solan 


TechnoRead 


0.664 


0.365 


0.134 


-0.053 


1.380 


1.816 


Underwood 


ILS 


-0.027 


0.174 


0.030 


-0.367 


0.314 


-0.153 


Vollands 


Edinburgh 


0.657 


0.352 


0.124 


-0.033 


1.347 


1.867 


Vollands 


Edinburgh B 


-0.472 


0.289 


0.084 


-1.038 


0.094 


-1.633 


Vollands 


Neale Ace 


0.326 


0.405 


0.164 


-0.468 


1.120 


0.805 


Vollands 


Neale Ace B 


0.150 


0.403 


0.162 


-0.639 


0.940 


0.373 


Vollands 


Neale Comp 


1.403 


0.452 


0.205 


0.517 


2.290 


3.102 


Vollands 


Neale Comp B 


0.178 


0.403 


0.163 


-0.612 


0.968 


0.441 


Xin 


ClozeFollow 


0.024 


0.227 


0.052 


-0.421 


0.470 


0.108 


Xin 


ClozePost 


0.368 


0.229 


0.052 


-0.081 


0.817 


1.606 


Xin 


CompFollow 


0.112 


0.227 


0.052 


-0.334 


0.558 


0.493 


Xin 


CompPost 


0.245 


0.228 


0.052 


-0.202 


0.692 


1.075 


Xin 


WordFollow 


0.359 


0.229 


0.052 


-0.090 


0.808 


1.566 


Xin 


WordPost 


0.475 


0.230 


0.053 


0.024 


0.927 


2.062 


Random 
Effects Model 


Weighted Mean 


0.489 


0.112 


0.013 


0.269 


0.709 


4.360 



* for Z= 1.96, p<. 05; for Z= 2.58, /?<.() 1 
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